2025-04-03 23:35:18,699 [ 657248 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths)
2025-04-03 23:35:18,699 [ 657248 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths)
2025-04-03 23:35:18,699 [ 657248 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths)
2025-04-03 23:35:18,699 [ 657248 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration  (runner:110, check_args_and_update_paths)
clickhouse_integration_tests_volume
Running pytest container as: 'docker run   --rm --name clickhouse_integration_tests_bfaisf --privileged --dns-search='.' --memory=30709030912 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="  -rfEps --run-id=2 --color=no --durations=0 test_s3_assume_role/test.py::test_using_assumed_creds  -vvv" altinityinfra/integration-tests-runner:cd6390247eca '.
Start tests
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration
configfile: pytest.ini
plugins: random-0.2, timeout-2.2.0, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
collecting ... collected 1 item

test_s3_assume_role/test.py::test_using_assumed_creds FAILED             [100%]

=================================== FAILURES ===================================
___________________________ test_using_assumed_creds ___________________________

started_cluster = <helpers.cluster.ClickHouseCluster object at 0x7fcdf23091b0>

    def test_using_assumed_creds(started_cluster):
        instance = started_cluster.instances["node1"]
    
        # Create some file in non public-accessible minio
>       instance.query(
            """
                INSERT INTO FUNCTION s3
                    (
                        'http://minio1:9001/root/test_assume.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSVWithNames'
                    )
                SELECT number as num, toString(number) as strnum FROM numbers(5);
            """
        )

test_s3_assume_role/test.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
helpers/cluster.py:3678: in query
    return self.client.query(
helpers/client.py:39: in wrap
    return func(self, *args, **kwargs)
helpers/client.py:79: in query
    ).get_answer()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <helpers.client.CommandRequest object at 0x7fcdf2392380>

    def get_answer(self):
        self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT)
        self.stdout_file.seek(0)
        self.stderr_file.seek(0)
    
        stdout = self.stdout_file.read().decode("utf-8", errors="replace")
        stderr = self.stderr_file.read().decode("utf-8", errors="replace")
    
        if (
            self.timer is not None
            and not self.process_finished_before_timeout
            and not self.ignore_error
        ):
            logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}")
            raise QueryTimeoutExceedException("Client timed out!")
    
        if (
            self.process.returncode != 0 or self.remove_trash_from_stderr(stderr)
        ) and not self.ignore_error:
>           raise QueryRuntimeException(
                "Client failed! Return code: {}, stderr: {}".format(
                    self.process.returncode, stderr
                ),
                self.process.returncode,
                stderr,
            )
E           helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 24.12.2):
E           Code: 499. DB::Exception: Received from 172.16.1.6:9000. DB::Exception: Failed to check existence of key test_assume.csv in bucket root: No response body.. Stack trace:
E           
E           0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d79979b
E           1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000868558c
E           2. DB::S3Exception::S3Exception<String const&, String const&, String const&>(Aws::S3::S3Errors, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String const&>::type, std::type_identity<String const&>::type>, String const&, String const&, String const&) @ 0x0000000010c562b2
E           3. DB::S3::objectExists(DB::S3::Client const&, String const&, String const&, String const&) @ 0x0000000010c56227
E           4. DB::S3ObjectStorage::exists(DB::StoredObject const&) const @ 0x0000000010c4897c
E           5. DB::checkAndGetNewFileOnInsertIfNeeded(DB::IObjectStorage const&, DB::StorageObjectStorage::Configuration const&, DB::StorageObjectStorage::QuerySettings const&, String const&, unsigned long) @ 0x0000000010b92bec
E           6. DB::StorageObjectStorage::write(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, bool) @ 0x0000000010afef49
E           7. DB::buildPushingToViewsChain(std::shared_ptr<DB::IStorage> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::Context const>, std::shared_ptr<DB::IAST> const&, unsigned long, bool, std::shared_ptr<DB::ThreadStatusesHolder>, std::shared_ptr<DB::ThreadGroup>, std::atomic<unsigned long>*, bool, DB::Block const&) @ 0x0000000013acfbe1
E           8. DB::InterpreterInsertQuery::buildSink(std::shared_ptr<DB::IStorage> const&, unsigned long, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::ThreadStatusesHolder>, std::shared_ptr<DB::ThreadGroup>, std::atomic<unsigned long>*) @ 0x0000000012058f96
E           9. DB::InterpreterInsertQuery::buildPreAndSinkChains(unsigned long, unsigned long, std::shared_ptr<DB::IStorage>, unsigned long, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Block const&) @ 0x0000000012059c1e
E           10. DB::InterpreterInsertQuery::buildInsertSelectPipeline(DB::ASTInsertQuery&, std::shared_ptr<DB::IStorage>) @ 0x000000001205bc95
E           11. DB::InterpreterInsertQuery::execute() @ 0x000000001205dff6
E           12. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000012433f35
E           13. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001242ed3d
E           14. DB::TCPHandler::runImpl() @ 0x000000001370c25e
E           15. DB::TCPHandler::run() @ 0x00000000137274d8
E           16. Poco::Net::TCPServerConnection::start() @ 0x000000001663d1c7
E           17. Poco::Net::TCPServerDispatcher::run() @ 0x000000001663d659
E           18. Poco::PooledThread::run() @ 0x0000000016609f7c
E           19. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001660851d
E           20. ? @ 0x00007f8b5e59bac3
E           21. ? @ 0x00007f8b5e62d850
E           . (S3_ERROR)
E           (query: INSERT INTO FUNCTION s3
E                           (
E                               'http://minio1:9001/root/test_assume.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSVWithNames'
E                           )
E                       SELECT number as num, toString(number) as strnum FROM numbers(5);)

helpers/client.py:248: QueryRuntimeException
---------------------------- Captured stdout setup -----------------------------
Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
------------------------------ Captured log setup ------------------------------
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : No running containers (conftest.py:96, cleanup_environment)
2025-04-03 23:35:22 [ 661 ] DEBUG : Pruning Docker networks (conftest.py:98, cleanup_environment)
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[docker network prune --force] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:146, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                  DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV         CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                             MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                        JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                             TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                 HOSTNAME 0ee5082ea25a (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                    SHLVL 0 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                     HOME /root (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                   OLDPWD / (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                        DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                         PYTHONUNBUFFERED 1 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                 DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                            UBSAN_OPTIONS print_stacktrace=1 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                           PYTEST_ADDOPTS   -rfEps --run-id=2 --color=no --durations=0 test_s3_assume_role/test.py::test_using_assumed_creds  -vvv (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV    CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                     COMPOSE_HTTP_TIMEOUT 600 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV              DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                 DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV         CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV               DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                     PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV             DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                           DOCKER_CHANNEL stable (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                    DOCKER_CLIENT_TIMEOUT 300 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV        DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                     DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV           DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                      PWD /ClickHouse/tests/integration (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV             DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV       CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV         CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                       TZ Etc/UTC (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                          DOCKER_BASE_TAG 6712d5cc610d (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                               SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                                 LC_CTYPE C.UTF-8 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                 INTEGRATION_TESTS_RUN_ID 2 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                        WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : ENV                      PYTEST_CURRENT_TEST test_s3_assume_role/test.py::test_using_assumed_creds (setup) (cluster.py:450, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:774, __init__)
2025-04-03 23:35:22 [ 661 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1729, add_instance)
2025-04-03 23:35:22 [ 661 ] DEBUG : Cluster name: project_name:roottests3assumerole. Added instance name:node1 tag:6712d5cc610d base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env', '--project-name', 'roottests3assumerole', '--file', '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2025, add_instance)
2025-04-03 23:35:22 [ 661 ] INFO : Starting cluster... (test.py:32, started_cluster)
2025-04-03 23:35:22 [ 661 ] INFO : Running tests in /ClickHouse/tests/integration/test_s3_assume_role/test.py (cluster.py:2793, start)
2025-04-03 23:35:22 [ 661 ] DEBUG : Cluster start called. is_up=False (cluster.py:2800, start)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker networks for project roottests3assumerole are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:873, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker containers for project roottests3assumerole are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:881, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker volumes for project roottests3assumerole are DRIVER    VOLUME NAME
 (cluster.py:889, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Cleanup called (cluster.py:894, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker networks for project roottests3assumerole are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:873, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker containers for project roottests3assumerole are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:881, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Docker volumes for project roottests3assumerole are DRIVER    VOLUME NAME
 (cluster.py:889, print_all_docker_pieces)
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3assumerole-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : No running containers for project: roottests3assumerole (cluster.py:922, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Images pruned (cluster.py:947, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check)
2025-04-03 23:35:22 [ 661 ] DEBUG : Volumes pruned: 1
 (cluster.py:958, cleanup)
2025-04-03 23:35:22 [ 661 ] DEBUG : Setup directory for instance: node1 (cluster.py:2813, start)
2025-04-03 23:35:22 [ 661 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4639, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Create directory for common tests configuration (cluster.py:4644, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Copy common configuration from helpers (cluster.py:4664, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Generate and write macros file (cluster.py:4716, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/configs/config.d (cluster.py:4752, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/database (cluster.py:4769, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/logs (cluster.py:4780, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4864, create_dir)
2025-04-03 23:35:22 [ 661 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_assume_role/_instances-2/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env (cluster.py:97, _create_env_file)
2025-04-03 23:35:22 [ 661 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2025-04-03 23:35:22 [ 661 ] DEBUG : No config file found (config.py:28, find_config_file)
2025-04-03 23:35:22 [ 661 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file)
2025-04-03 23:35:22 [ 661 ] DEBUG : No config file found (config.py:28, find_config_file)
2025-04-03 23:35:23 [ 661 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request)
2025-04-03 23:35:23 [ 661 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --project-name roottests3assumerole --file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml pull] (cluster.py:122, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: proxy2 Skipped - Image is already being pulled by proxy1  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: minio1 Pulling  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: proxy1 Pulling  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: node1 Pulling  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: resolver Pulling  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: proxy1 Pulled  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: resolver Pulled  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: node1 Pulled  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] DEBUG : Stderr: minio1 Pulled  (cluster.py:148, run_and_check)
2025-04-03 23:35:33 [ 661 ] INFO : Trying to create Minio instance by command docker compose --project-name roottests3assumerole --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:3132, start)
2025-04-03 23:35:33 [ 661 ] DEBUG : Command:[docker compose --project-name roottests3assumerole --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] (cluster.py:122, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr:time="2025-04-03T23:35:33Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Network roottests3assumerole_default  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Network roottests3assumerole_default  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Volume "roottests3assumerole_data1-1"  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Volume "roottests3assumerole_data1-1"  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: proxy1 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested  (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: proxy2 The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested  (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Starting (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Starting (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Started (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Started (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Starting (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Starting (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Started (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Started (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr:time="2025-04-03T23:35:34Z" level=debug msg="otel error" error="<nil>" (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] DEBUG : Stderr:time="2025-04-03T23:35:34Z" level=debug msg="otel error" error="<nil>" (cluster.py:148, run_and_check)
2025-04-03 23:35:34 [ 661 ] INFO : Trying to connect to Minio... (cluster.py:3138, start)
2025-04-03 23:35:34 [ 661 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2082, get_instance_ip)
2025-04-03 23:35:34 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3assumerole-minio1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:34 [ 661 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2082, get_instance_ip)
2025-04-03 23:35:34 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3assumerole-proxy1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:34 [ 661 ] DEBUG : Starting new HTTP connection (1): 172.16.1.4:9001 (connectionpool.py:245, _new_conn)
2025-04-03 23:35:34 [ 661 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2025-04-03 23:35:34 [ 661 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcdf2390550>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen)
2025-04-03 23:35:34 [ 661 ] DEBUG : Starting new HTTP connection (2): 172.16.1.4:9001 (connectionpool.py:245, _new_conn)
2025-04-03 23:35:34 [ 661 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2025-04-03 23:35:34 [ 661 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcdf2390790>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen)
2025-04-03 23:35:34 [ 661 ] DEBUG : Starting new HTTP connection (3): 172.16.1.4:9001 (connectionpool.py:245, _new_conn)
2025-04-03 23:35:34 [ 661 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment)
2025-04-03 23:35:34 [ 661 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcdf23909d0>: Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen)
2025-04-03 23:35:34 [ 661 ] DEBUG : Starting new HTTP connection (4): 172.16.1.4:9001 (connectionpool.py:245, _new_conn)
2025-04-03 23:35:34 [ 661 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.1.4', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcdf2390c10>: Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2637, wait_minio_to_start)
2025-04-03 23:35:35 [ 661 ] DEBUG : Starting new HTTP connection (5): 172.16.1.4:9001 (connectionpool.py:245, _new_conn)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://172.16.1.4:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : Connected to Minio. (cluster.py:2617, wait_minio_to_start)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://172.16.1.4:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://172.16.1.4:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : S3 bucket 'root' created (cluster.py:2632, wait_minio_to_start)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://172.16.1.4:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://172.16.1.4:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : S3 bucket 'root2' created (cluster.py:2632, wait_minio_to_start)
2025-04-03 23:35:35 [ 661 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --project-name roottests3assumerole --file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml up -d --no-recreate') (cluster.py:3200, start)
2025-04-03 23:35:35 [ 661 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --project-name roottests3assumerole --file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml up -d --no-recreate] (cluster.py:122, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Running (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Running (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Running (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Running (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Creating (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Created (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Starting (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Started (cluster.py:148, run_and_check)
2025-04-03 23:35:35 [ 661 ] DEBUG : ClickHouse instance created (cluster.py:3208, start)
2025-04-03 23:35:35 [ 661 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2082, get_instance_ip)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3assumerole-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2092, get_instance_global_ipv6)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3assumerole-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.1.6... (cluster.py:3216, start)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3assumerole-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/5bc4043387c5e826ef6f0bcb7e4c1e3691f3e9d1f1de7eba276289288784cae8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/5bc4043387c5e826ef6f0bcb7e4c1e3691f3e9d1f1de7eba276289288784cae8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:35 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/5bc4043387c5e826ef6f0bcb7e4c1e3691f3e9d1f1de7eba276289288784cae8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:36 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/5bc4043387c5e826ef6f0bcb7e4c1e3691f3e9d1f1de7eba276289288784cae8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:36 [ 661 ] DEBUG : http://localhost:None "GET /v1.46/containers/5bc4043387c5e826ef6f0bcb7e4c1e3691f3e9d1f1de7eba276289288784cae8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request)
2025-04-03 23:35:36 [ 661 ] DEBUG : ClickHouse node1 started (cluster.py:3220, start)
2025-04-03 23:35:36 [ 661 ] INFO : Starting mock server mock_sts.py (mock_servers.py:18, start_mock_servers)
2025-04-03 23:35:36 [ 661 ] DEBUG : run container_id:roottests3assumerole-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKTU9DS19YTUxfUkVTUE9OU0UgPSAiIiI8QXNzdW1lUm9sZVJlc3BvbnNlIHhtbG5zPSJodHRwczovL3N0cy5hbWF6b25hd3MuY29tL2RvYy8yMDExLTA2LTE1LyI+CiAgPEFzc3VtZVJvbGVSZXN1bHQ+CiAgICA8Q3JlZGVudGlhbHM+CiAgICAgIDxBY2Nlc3NLZXlJZD5taW5pbzwvQWNjZXNzS2V5SWQ+CiAgICAgIDxTZWNyZXRBY2Nlc3NLZXk+Q2xpY2tIb3VzZV9NaW5pb19QQHNzdzByZDwvU2VjcmV0QWNjZXNzS2V5PgogICAgICA8RXhwaXJhdGlvbj4yMDU1LTEyLTMxVDIzOjU5OjU5WjwvRXhwaXJhdGlvbj4KICAgIDwvQ3JlZGVudGlhbHM+CiAgPC9Bc3N1bWVSb2xlUmVzdWx0Pgo8L0Fzc3VtZVJvbGVSZXNwb25zZT4iIiIKCiMgPFNlc3Npb25Ub2tlbj5NT0NLX1NFU1NJT05fVE9LRU48L1Nlc3Npb25Ub2tlbj4gIGlzIG5vdCBiZWluZyByZXR1cm5lZCAtLSBpdCBpcyBub3QgcmVxdWlyZWQgYnkgbWluaW8KIyBXaGVuICJyZWFsIiBTVFMgcmV0dXJucyBpdCAtLSBpdCBpcyBhbHNvIHVzZWQgdG8gcmVhZCBmcm9tIHMzCgpAcm91dGUoIi8iLCBtZXRob2Q9IlBPU1QiKQpkZWYgcmV0dXJuX2NyZWRzKCk6CiAgICByZXNwb25zZS5zdGF0dXMgPSAyMDAKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJhcHBsaWNhdGlvbi94bWwiCiAgICByZXR1cm4gTU9DS19YTUxfUkVTUE9OU0UKCgpAcm91dGUoIi8iLCBtZXRob2Q9IkdFVCIpCmRlZiBwaW5nKCk6CiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQ== | base64 --decode > mock_sts.py'] (cluster.py:2126, exec_in_container)
2025-04-03 23:35:36 [ 661 ] DEBUG : Command:[docker exec roottests3assumerole-resolver-1 bash -c echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKTU9DS19YTUxfUkVTUE9OU0UgPSAiIiI8QXNzdW1lUm9sZVJlc3BvbnNlIHhtbG5zPSJodHRwczovL3N0cy5hbWF6b25hd3MuY29tL2RvYy8yMDExLTA2LTE1LyI+CiAgPEFzc3VtZVJvbGVSZXN1bHQ+CiAgICA8Q3JlZGVudGlhbHM+CiAgICAgIDxBY2Nlc3NLZXlJZD5taW5pbzwvQWNjZXNzS2V5SWQ+CiAgICAgIDxTZWNyZXRBY2Nlc3NLZXk+Q2xpY2tIb3VzZV9NaW5pb19QQHNzdzByZDwvU2VjcmV0QWNjZXNzS2V5PgogICAgICA8RXhwaXJhdGlvbj4yMDU1LTEyLTMxVDIzOjU5OjU5WjwvRXhwaXJhdGlvbj4KICAgIDwvQ3JlZGVudGlhbHM+CiAgPC9Bc3N1bWVSb2xlUmVzdWx0Pgo8L0Fzc3VtZVJvbGVSZXNwb25zZT4iIiIKCiMgPFNlc3Npb25Ub2tlbj5NT0NLX1NFU1NJT05fVE9LRU48L1Nlc3Npb25Ub2tlbj4gIGlzIG5vdCBiZWluZyByZXR1cm5lZCAtLSBpdCBpcyBub3QgcmVxdWlyZWQgYnkgbWluaW8KIyBXaGVuICJyZWFsIiBTVFMgcmV0dXJucyBpdCAtLSBpdCBpcyBhbHNvIHVzZWQgdG8gcmVhZCBmcm9tIHMzCgpAcm91dGUoIi8iLCBtZXRob2Q9IlBPU1QiKQpkZWYgcmV0dXJuX2NyZWRzKCk6CiAgICByZXNwb25zZS5zdGF0dXMgPSAyMDAKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJhcHBsaWNhdGlvbi94bWwiCiAgICByZXR1cm4gTU9DS19YTUxfUkVTUE9OU0UKCgpAcm91dGUoIi8iLCBtZXRob2Q9IkdFVCIpCmRlZiBwaW5nKCk6CiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQ== | base64 --decode > mock_sts.py] (cluster.py:122, run_and_check)
2025-04-03 23:35:36 [ 661 ] DEBUG : run container_id:roottests3assumerole-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 mock_sts.py 8081 >/var/log/resolver/mock_sts.log 2>/var/log/resolver/mock_sts.err.log'] (cluster.py:2126, exec_in_container)
2025-04-03 23:35:36 [ 661 ] DEBUG : Command:[docker exec roottests3assumerole-resolver-1 bash -c python3 mock_sts.py 8081 >/var/log/resolver/mock_sts.log 2>/var/log/resolver/mock_sts.err.log] (cluster.py:122, run_and_check)
2025-04-03 23:35:36 [ 661 ] DEBUG : run container_id:roottests3assumerole-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2126, exec_in_container)
2025-04-03 23:35:36 [ 661 ] DEBUG : Command:[docker exec roottests3assumerole-resolver-1 curl -s http://localhost:8081/] (cluster.py:122, run_and_check)
2025-04-03 23:35:36 [ 661 ] DEBUG : Exitcode:7 (cluster.py:150, run_and_check)
2025-04-03 23:35:37 [ 661 ] DEBUG : run container_id:roottests3assumerole-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] (cluster.py:2126, exec_in_container)
2025-04-03 23:35:37 [ 661 ] DEBUG : Command:[docker exec roottests3assumerole-resolver-1 curl -s http://localhost:8081/] (cluster.py:122, run_and_check)
2025-04-03 23:35:37 [ 661 ] DEBUG : Stdout:OK (cluster.py:146, run_and_check)
2025-04-03 23:35:37 [ 661 ] DEBUG : mock_sts.py answered OK on attempt 2 (mock_servers.py:67, start_mock_servers)
2025-04-03 23:35:37 [ 661 ] INFO : Mock server mock_sts.py started (mock_servers.py:82, start_mock_servers)
------------------------------ Captured log call -------------------------------
2025-04-03 23:35:37 [ 661 ] DEBUG : Executing query 
            INSERT INTO FUNCTION s3
                (
                    'http://minio1:9001/root/test_assume.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSVWithNames'
                )
            SELECT number as num, toString(number) as strnum FROM numbers(5);
         on node1 (cluster.py:3677, query)
---------------------------- Captured log teardown -----------------------------
2025-04-03 23:35:37 [ 661 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --project-name roottests3assumerole --file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml stop --timeout 20] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/.env --project-name roottests3assumerole --file /ClickHouse/tests/integration/test_s3_assume_role/_instances-2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml down --volumes] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-node1-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-resolver-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-minio1-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Stopping (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Stopped (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy2-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Container roottests3assumerole-proxy1-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Volume roottests3assumerole_data1-1  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Volume roottests3assumerole_data1-1  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Network roottests3assumerole_default  Removing (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stderr: Network roottests3assumerole_default  Removed (cluster.py:148, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Cleanup called (cluster.py:894, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Docker networks for project roottests3assumerole are NETWORK ID   NAME      DRIVER    SCOPE
 (cluster.py:873, print_all_docker_pieces)
2025-04-03 23:35:58 [ 661 ] DEBUG : Docker containers for project roottests3assumerole are CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
 (cluster.py:881, print_all_docker_pieces)
2025-04-03 23:35:58 [ 661 ] DEBUG : Docker volumes for project roottests3assumerole are DRIVER    VOLUME NAME
 (cluster.py:889, print_all_docker_pieces)
2025-04-03 23:35:58 [ 661 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3assumerole-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : No running containers for project: roottests3assumerole (cluster.py:922, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Images pruned (cluster.py:947, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup)
2025-04-03 23:35:58 [ 661 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Stdout:1 (cluster.py:146, run_and_check)
2025-04-03 23:35:58 [ 661 ] DEBUG : Volumes pruned: 1
 (cluster.py:958, cleanup)
============================== slowest durations ===============================
21.23s teardown test_s3_assume_role/test.py::test_using_assumed_creds
14.51s setup    test_s3_assume_role/test.py::test_using_assumed_creds
0.06s call     test_s3_assume_role/test.py::test_using_assumed_creds
=========================== short test summary info ============================
FAILED test_s3_assume_role/test.py::test_using_assumed_creds - helpers.client...
============================== 1 failed in 35.97s ==============================
Traceback (most recent call last):
  File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in <module>
    subprocess.check_call(cmd, shell=True)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'docker run   --rm --name clickhouse_integration_tests_bfaisf --privileged --dns-search='.' --memory=30709030912 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e   -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600  -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="  -rfEps --run-id=2 --color=no --durations=0 test_s3_assume_role/test.py::test_using_assumed_creds  -vvv" altinityinfra/integration-tests-runner:cd6390247eca ' returned non-zero exit status 1.